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Abstract — The Hierarchical SEGmentation (HSEG) algorithm, 
which combines region object finding with region object cluster- 
ing, has given good performances for multi- and hyperspectral 
image analysis. This technique produces at its output a hier- 
archical set of image segmentations. The automated selection 
of a single segmentation level is often necessary. We propose 
and investigate the use of automatically selected markers for this 
purpose. In this paper, a novel Marker-based HSEG (M-HSEG) 
method for spectral-spatial classification of hyperspectral images 
is proposed. Two classification-based approaches for automatic 
marker selection are adapted and compared for this purpose. 
Then, a novel constrained marker-based HSEG algorithm is 
applied, resulting in a spectral-spatial classification map. Three 
different implementations of the M-HSEG method are proposed 
and their performances in terms of classification accuracies 
are compared. The experimental results, presented for three 
hyperspectral airborne images, demonstrate that the proposed 
approach yields accurate segmentation and classification maps, 
and thus is attractive for remote sensing image analysis. 

Index Terms — Hierarchical segmentation, classification, 

marker selection, hyperspectral images. 

I. Introduction 

One of the most challenging problems currently addressed 
by the remote sensing community is the development of 
effective analysis and processing techniques for data acquired 
with the last generation of imaging sensors [1]. In particu- 
lar, hyperspectral imaging is a relatively new technique that 
records the energy of the received light in tens or hundreds 
of narrow spectral bands in each spatial position in the 
image [2]. Therefore, it becomes possible to identify physical 
materials and classify regions within the image scene with 
much higher accuracies when compared to panchromatic or 
multispectral sensors. Remote sensing image classification, 
which can be defined as identification of objects in a scene 
captured by a remote imaging sensor, is an important task 
in many application domains such as precision agriculture, 
monitoring of environment, urban planning, etc. 

Most of the previously proposed classification methods 
process each pixel independently using its spectral values 
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only [3], [4], [5], [6]. One of the most frequently used 
techniques is Support Vector Machines (SVM) [7], [8]. Recent 
studies have shown the advantage of including information 
about spatial dependencies for accurate image analysis, i.e., 
performing spectral- spatial classification [9], [10], [11], [12]. 

In the seminal works on spectral- spatial image classification, 
the information from the closest neighborhoods, defined by 
either fixed windows [9], [13] or morphological profiles [14], 
has been considered for classifying each pixel. In our recent 
works, we have shown the advantage of using segmentation for 
distinguishing spatial structures in a hyperspectral image [15], 
[16]. Segmentation can be defined as an exhaustive partitioning 
of the input image into homogeneous regions (with respect to 
some criterion of interest, e.g., intensity or texture) [17], [18]. 
Different techniques have been investigated for segmentation 
of hyperspectral data, such as watershed, partitional clustering, 
and Hierarchical SEGmentation (HSEG). Then, each region 
from a segmentation map was classified by applying a majority 
vote rule over the pixelwise SVM classification results [16]. 

The HSEG method is one of the few available segmen- 
tation approaches in the state-of-art that naturally integrates 
spatial and spectral information. HSEG is a combination of 
region object finding by hierarchical step-wise optimization 
(HSWO, or iterative best merge region growing) [19] and 
region clustering by grouping spectrally similar but spatially 
disjoint regions [20], [21]. Unlike most other segmentation 
approaches, the HSEG produces at its output a segmentation 
hierarchy. A segmentation hierarchy is defined as a set of 
image segmentations at different levels of detail, in which seg- 
mentations at coarser levels of details are produced by merging 
regions at finer levels of detail. In this hierarchy, an object of 
interest may be represented by several regions at finer levels of 
detail, and may be merged with another region at coarser levels 
of detail. It is often necessary to choose a single optimum 
hierarchical segmentation level, which depends on the specific 
application (e. g., the specific classification problem). For 
instance, when performing urban data analysis, a coarser 
segmentation map may divide an image into impervious/non- 
impervious surfaces, while a finer segmentation map may be 
composed of regions representing individual buildings, trees, 
etc. In [16], we have selected an appropriate level of seg- 
mentation detail for spectral- spatial classification 1 interactively 

1 In [16], we verified that the interactively selected hierarchical level 
was optimal by quantitatively evaluating the segmentation results at several 
hierarchical levels versus the test data and retaining the best results. 
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with the program HSEGViewer [21]. However, an automatic 
procedure would be desirable. Plaza and Tilton [22] proposed 
to use joint spectral/spatial homogeneity scores computed from 
the segmented regions, for automating the selection of relevant 
hierarchical level(s). This approach may be computationally 
expensive for large data sets. 

The main focus of this paper is to propose a method for the 
automated selection of a single hierarchical segmentation level. 
This objective can be achieved by incorporation of some addi- 
tional knowledge into a segmentation procedure. We propose 
and investigate the use of automatically derived markers , or 
region seeds, for this purpose. A marker in our study is defined 
as a set of image pixels (which are not necessarily spatially 
connected) associated with one object in the image scene. The 
marker-controlled segmentation approach determines a marker 
for each region of interest and then segments an image in such 
a way that each region in a segmentation map contains one 
marker. In order to accurately segment an image, a marker for 
each image object/region must be selected. 

The problem of automatic marker selection has been dis- 
cussed in previous studies. Markers are often chosen by 
searching flat zones (i.e., connected components of pixels of 
constant gray-level value), zones of homogeneous texture, or 
image extrema [23]. Gomez et al. [24] used histogram analysis 
to obtain a set of representative pixel values, and the markers 
were generated with all the image pixels having representative 
gray values. Noyel et al. [25], [26] performed classification 
of the hyperspectral image (using different methods, such as 
Clara [27] and linear discriminant analysis [28]) and then 
filtered the classification maps class by class, using mathemat- 
ical morphology operators, for selecting large spatial regions 
as markers. In [12], we have proposed to use probability 
estimates obtained by the pixelwise SVM classification in 
order to choose the most reliably classified pixels as markers 
of spatial regions. Furthermore, a Minimum Spanning Forest 
(MSF) rooted on the selected markers was built, resulting in 
a spectral- spatial classification map. One of the advantages 
of applying classification-based methods for marker selection 
is that segmentation regions grown from these markers can 
be immediately assigned to the class of the corresponding 
classification-derived marker, thus yielding a classification 
map. 

In this paper we propose a new Marker-based HSEG (M- 
HSEG) method for segmentation and spectral- spatial clas- 
sification of hyperspectral images. Two classification-based 
approaches for marker selection proposed in [12] and [26] 
are adapted and compared in order to define relevant markers 
for the HSEG procedure. First, markers of spatial regions 
are automatically selected using classification results. Then, 
a novel constrained M-HSEG algorithm is applied, resulting 
in both segmentation and classification maps. We propose and 
discuss several ways of integrating markers into the HSEG 
algorithm. 

Although the proposed marker-based segmentation scheme 
was designed for hyperspectral data, the method is general 
and can be applied to other types of data as well. Ex- 
perimental results are demonstrated on three hyperspectral 
airborne images acquired by the Reflective Optics System 



Fig. 1. Flow-chart of the proposed M-HSEG classification scheme. 

Imaging Spectrometer (ROSIS), the Airborne Visible/Infrared 
Imaging Spectrometer (AVIRIS), and the Airborne Imaging 
System for different Applications (AISA) Eagle Sensor. The 
proposed M-HSEG approach is compared with the recently 
proposed marker-based classification technique, which consists 
in constructing an MSF from a set of markers [12], as well as 
with several other spectral- spatial classification techniques. 

The outline of this paper is as follows. The next sec- 
tion describes two classification-based approaches for marker 
selection adapted for the proposed M-HSEG scheme. In 
Section III, the proposed constrained M-HSEG method and 
its implementations are presented. Section IV presents an 
MSF algorithm used for comparison. Experimental results 
are discussed in Section V and conclusions are drawn in 
Section VI. 

II. Classification-based marker selection 

The flow-chart of the proposed M-HSEG classification 
method is depicted in Fig. 1. An input 5-band hyperspec- 
tral image can be considered as a set of n pixel vectors 
X = {xj G j = 1, 2, ..., n}. Let Q = {cci, CC 2 , be 

a set of information classes in the image scene. Classification 
consists in assigning each pixel to one of the K classes of 
interest. 

Because in our study markers are automatically selected us- 
ing classification results, the preliminary step for both marker 
selection techniques consists in performing a pixelwise clas- 
sification. In the following, two marker selection techniques 
adapted for the M-HSEG method are described. 

A. Morpho-MS approach 

The Morpho-MS marker selection technique follows the 
approach proposed in [26], which is based on morphological 
filtering of the classification map. The proposed method is 
based on an assumption that if a pixelwise classifier has 
assigned a large set of spatially connected pixels to the same 
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class, the corresponding region is relevant and must contain a 
marker. The Morpho-MS technique consists of the following 
steps: 


1) Perform a pixelwise classification of a hyperspectral 
image. We propose to use an SVM classifier for this 
purpose, which is very well suited to classify high- 
dimensional data when a limited number of training 
samples is available [7], [29]. We refer the reader to [7], 
[30] for details on the SVM technique. This step results 
in a classification map L = {Ljj = 1, 2, ...,n}, where 
each pixel Xj has a class label Lj. 

2) Generate K binary images h Uk ( k = 1, ..., K) associated 
with K classes, so that pixels assigned to the class 
have the value k in the image for the corresponding 
class: 


K k {j) = 


k, 

o, 


if Lj = k ; 

otherwise. 


( 1 ) 


3) Perform morphological erosion of each image h Uk by a 
preset structuring element (SE) N corresponding to the 
pixel neighborhood. The erosion assigns a zero value to 
the pixels situated on the boundaries of objects in the 
image scene (assuming that the uncertainty of correct 
classification on the boundaries is higher than in the 
center of objects). The remaining non-zero pixels corre- 
spond to the cores (centers) of large spatial structures, 
which are assumed to be reliably classified and therefore 
can be used as region markers. The choice of an SE 
depends on the size of objects of interest in the image. 
We propose to use an elementary 3x3 square SE, 
in order to minimize the risk of loosing small spatial 
structures when selecting markers. 

4) Combine K images h' Uk using the pixelwise maximum 
operator: 

0(j)m k ma^ K {h' Uk (j)). (2) 


5) Perform a connected component labeling of C, using an 
eight-neighborhood connectivity [31]. 

6) Finally, select each connected component with non-zero 
values ( k ) in L' as a marker, with the corresponding 
class label ( k ). 

The resulting marker map contains markers (spatially con- 
nected sets of pixels) for large, i.e., reliably classified regions. 
The drawback of the Morpho-MS technique is that it is not 
able to define a marker for a spatial object which is smaller 
than the SE. Therefore, small objects in the image risk to be 
assimilated with their neighboring structures. 


B. Proha-MS approach 

A second marker selection technique, Proha-MS (proposed 
in [12]), mitigates the drawback of the Morpho-MS approach, 
by analyzing both the sizes of connected components in the 
classification map and the probability estimates of correct 
classification. The Proha-MS method consists of two steps: 

1) Prohahilistic pixelwise classification: Perform a proba- 
bilistic pixelwise SVM classification of a hyperspectral im- 
age [8], [30]. This step results in a classification map (where 


each pixel has a unique class label) and a probability map 
(containing probability estimates for each pixel to belong to 
the assigned class). 

Standard SVM classifications do not provide probability 
estimates for the individual classes. In order to get these esti- 
mates, pairwise coupling of binary probability estimates can be 
applied [8], [32]. In our research we use the probabilistic SVM 
method implemented in LIB SVM library [8]. The objective is 
to estimate, for each pixel x, classification probabilities: 

P = {Pk=p(L x = k\x),k=l,...,K}. (3) 

For this purpose, first pairwise class probabilities « 
p(L x = i\L x = i or j,x) are estimated, and then the 
probabilities in (3) are computed, as described in [32]. Finally, 
a probability map is constructed, by assigning to each pixel 
the maximum probability estimate ma,x(pk),k = 1, 

2) Marker selection: Apply a connected component labeling 
on the classification map, using an eight-neighborhood con- 
nectivity [31]. Then, analyze each connected component as 
follows: 

• if a region is large (number of pixels in the region > M), 
it is considered to be relevant, and its marker is defined 
as the P% of pixels within this region with the highest 
probability estimates. 

• If a region is small , it is further investigated if its 
pixels were classified to a particular class with a high 
probability. Its potential marker is formed by the pixels 
with probability estimates higher than a defined threshold 

S. 

The procedure of the choice of parameters (M, P, S ) for the 
Proha-MS technique is described in [12]. At the output of the 
marker selection step, a map of m markers is obtained, where 
each marker O* = {xj e X, j = 1, 2, ..., card(Oi); Lo*} 
(i = 1, ..., mn) consists of one or several pixels ( card(Oi ) = 1 
or card(Oi ) > 1, respectively) and has a class label Lq.. 
One should note that a marker is not necessarily a spatially 
connected set of pixels. 

III. Marker-based HSEG 

A key feature of the HSEG algorithm is the tight inter- 
twining of region growing segmentation (using the HSWO 
approach [19]), which produces spatially connected regions, 
with region clustering, which groups together similar spatially 
disjoint regions [20], [21]. The following outline of HSEG 
summarizes the description given in [21]: 

Initialization: Initialize the segmentation by assigning each 
image pixel a region label. If a pre- segmentation is provided, 
label each pixel according to the presegmentation. Otherwise, 
label each pixel as a separate region. 

1) Calculate the Dissimilarity Criterion (DC) value between 
all pairs of spatially adjacent regions. A spatially ad- 
jacent region for a given region is the one containing 
pixels situated in the neighborhood (eight-neighborhood 
is used in our study) of the given region’s pixels. 

2) Find the smallest DC value dissim_val and set 
thresh jual equal to it. Then, merge all pairs of spatially 
adjacent regions with dissim_val = threshjual. 
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Fig. 2. Scheme illustrating the M-HSEG algorithm. 


3) If the parameter S wg ht > 0.0, merge all pairs of spatially 
non-adjacent regions with 

dissim_val < S wg ht • threshjval. 

4) Stop if no more merges are required (convergence is 
achieved). Otherwise, return to step 1. 

Different measures can be applied for computing DCs 
between regions, such as vector norms and Spectral Angle 
Mapper (SAM) between the region mean vectors [21]. The 
LI vector norm between two vectors = (un, ..., uib) t and 
xx j = (uj i, Ujs) T is defined as 

B 

Ll(ui,Uj) = ^2\u ib - u jb \. (4) 

6=1 

The vector Infinity (Inf) norm (oc-norm) between and 
Uj is given as 


Inf(ui,Uj) = ||uj — UjHoo = max \u ib - u jb \. (5) 

o=l,..., B 


The SAM measure between and u j determines the 
spectral similarity between two vectors by computing the angle 
between them. It is computed as 


SAM(\Xi. Uj) = arccos 


'Yhb= 1 u ib u jb 

,Ef =1 ^] 1/2 E6 B =i^ 2 6] 1/2 , 


( 6 ) 


The optional parameter S wg ht , in HSEG, tunes the relative 
importance of clustering based on spectral information only 
versus region growing. If S wg ht = 0 - 0 , only spatially adjacent 
regions are allowed to merge. If 0.0 < S wg ht < 1.0, 
spatially adjacent merges are favored compared with spatially 
nonadjacent merges by a factor of 1.0/ S wg ht- 

The allowance for the merging of spatially non-adjacent 
regions in HSEG leads to heavy computational demands. In 
order to reduce these demands, a recursive divide-and conquer 
approximation of HSEG (RHSEG) has been developed. The 
NASA Goddard RHSEG software provides an efficient parallel 
implementation of the RHSEG algorithm. 

HSEG naturally produces a segmentation hierarchy consist- 
ing of a set of segmentations from initialization down to the 
final trivial one region segmentation (if allowed to proceed 
that far). However, for practical applications a subset of one 
or several segmentations needs to be selected out from this 
hierarchy. The proposed M-HSEG algorithm is designed for 
automatically selecting the segmentation level of detail for the 
most accurate segmentation and classification result. 


A. M-HSEG algorithm 

The main idea behind the marker-based HSEG algorithm 
consists in assigning a marker label for each region contain- 
ing marker pixels, and merging regions with an additional 
condition: two regions with different marker labels can not 
be merged together (see Fig. 2). The proposed M-HSEG 
algorithm can be summarized as follows: 

1) Initialize the segmentation by labeling either the whole 
marker (i.e., all the pixels belonging to the same marker), 
or an individual non-marker pixel as one region. Assign 
for every region the corresponding marker label (which 
is equal to zero for non-marked regions). 

2) At each iteration, perform HSEG, with an additional 
condition: two regions with different non-zero marker 
labels have the DC equal to infinity (in practice, the 
upper maximum value of float ) and are never merged 
together. When a marked region is merged with a non- 
marked region, the resulting region keeps the marker 
label inherited from the marked region. 

3) Stop the iterative process when either no more merging 
is possible (number of regions is equal to the number 
of markers) or the smallest DC between any two neigh- 
boring regions is higher than the preset (or computed) 
threshold. 

Classification : Assign the class of each marker to all pixels 
in the region containing this marker. If the segmentation result 
contains non-marked regions (i.e., M-HSEG converged before 
the number of regions was equal to the number of markers), 
classify these regions. For instance, this can be done by 
applying a majority vote rule over the pixel wise classification 
results [16]: for every non-marked region, all the pixels are 
assigned to the most frequent class within this region. 

B. Implementations of the M-HSEG algorithm 

We have investigated the performance of three different 
implementations of the proposed M-HSEG method. One im- 
plementation, M-HSEG r , is based on the description given in 
the previous subsection. All the pixels belonging to the same 
marker are initialized as one region, and then iterative region 
merging is performed. 

A second implementation, M-HSEG P , first initializes each 
pixel as one region and assigns a marker label for every 
region (equal to zero for non-marked regions). Thus, after 
the initialization, multiple regions can have the same non-zero 
marker label. During the region merging procedure, the regions 
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Fig. 3. Scheme illustrating region and marker labeling of marker pixels 
at the initialization step of three different implementations of the M-HSEG 
algorithm: initial markers 1 and 2 are denoted by red and green colors, 
respectively. 
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Fig. 4. Example of addition of extra vertices ti,t 2 ,r to the image graph 
for construction of the MSF rooted on markers 1 and 2; non-marker pixels 
are denoted by “0”. 

Each image pixel is considered as a vertex v G V of an 
undirected graph G = (V,E,W), where V and E are the 
sets of vertices and edges, respectively, and W is a weighting 
function. Each edge G E of this graph connects a couple 
of vertices i and j corresponding to the neighboring pixels 
(we assume eight-neighborhood in our study). A weight Wij 
is assigned to each edge which indicates the degree of 
dissimilarity between two vertices (i.e., two corresponding pix- 
els) connected by this edge. Different dissimilarity measures 
can be used for computing weights of edges, such as vector 
norms and SAM between two pixel vectors. 

Given a graph G = (V,E,W), a spanning forest F = 
(V, Ep) of G is a non-connected graph without cycles such 
that Ep C E. The MSF rooted on a set of m distinct vertices 
{ti , ...,£ m } is defined as a spanning forest F* = (V,Ep*) of 
G, such that each tree of F* is grown from one root £ z , and 
the sum of the edges weights of F* is minimal [33]: 


with equal non-zero marker labels have a zero DC value, while 
the regions with different non-zero marker labels have the DC 
equal to infinity. 

A third implementation, M-HSEG 025 , first initializes each 
pixel as one region. If the given pixel is marked, the corre- 
sponding region obtains a new non-zero marker label, with 
the corresponding information class. Thus, at the initialization 
step all the markers are split into one-pixel markers. Then, 
iterative region merging is performed, providing that regions 
with different markers cannot be merged together. At the final 
step, the regions containing pixels of the same initial marker 
are merged together. 

Fig. 3 illustrates a schematic example of region and marker 
labeling of marker pixels at the initialization step of the three 
proposed implementations of M-HSEG. The implementations 
M-HSEG 25 and M-HSEG 025 are useful when images contain 
large regions with high intra-region spectral variation. In this 
case, it may be advantageous to compute region feature vectors 
over parts of these regions. 

IV. Construction of an MSF 

This section recapitulates the marker-based classification 
technique recently proposed in [12] for hyperspectral im- 
age analysis, used for comparison with the novel M-HSEG 
method. It consists in constructing an MSF rooted on a set of 
markers. 


F* G arg min 
FeSF 



jeE F 


(7) 


where SF is a set of all spanning forests of G rooted on 
\t\ , fra} • 

In order to obtain the MSF rooted on markers, m additional 
vertices t i: i = 1 are introduced. Each extra vertex ti 

is connected by the edge with a null weight to the pixels 
representing a marker O^. Furthermore, an additional root 
vertex r is added and is connected by the null-weight edges 
to the vertices ti (see an example on figure 4). The minimum 
spanning tree [33] of the built graph induces an MSF in G, 
where each tree is grown on a vertex t L . The MSF is obtained 
after removing the vertex r. Prim’s algorithm can be used for 
building the MSF (we refer the readers to [12], [34] for details 
on this algorithm). 

Each tree in the MSF corresponds to a region in the segmen- 
tation map. A classification map is obtained by assigning the 
class of each marker to all the pixels grown from this marker. 


V. Experimental results 

Three hyperspectral data sets were used for the experiments, 
with different contexts (one urban area and two agricultural 
areas) and acquired by different sensors (ROSIS, AVIRIS and 
AISA Eagle) with different spectral and spatial resolutions. 
These data sets and the corresponding results are presented in 
the next three sections. 
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TABLE I 

Results for the Center of Pavia data set. Information Classes, Number of Labeled Samples and Classification Accuracies in 
Percentage, Using the Proba-MS Marker Selection Approach: Overall Accuracy (OA), Average Accuracy (AA), Kappa 
Coefficient (k) and Class-Specific Accuracies. The highest accuracies are bolded in each category. 



Number 
of labeled 
samples 

M-HSEG r 

M-HSEG P 

M-HSEG op 


MSF 

DC 

LI 

SAM 

LI 

SAM 

LI 

| SAM 

SAM 

SAM 

SVM 

LI 

SAM 

^wght 

0.0 

0.2 

0.5 

OA 


90.44 

91.32 

92.44 

92.86 

95.96 

96.20 

96.35 

95.50 

94.96 

96.74 

91.31 

AA 


85.69 

88.03 

89.59 

90.43 

95.00 

95.26 

95.49 

94.18 

92.56 

95.66 

92.64 

K 


87.90 

89.02 

90.43 

90.96 

94.88 

95.18 

95.37 

94.30 

93.61 

95.86 

89.11 

Water 

12734 

97.41 

97.80 

97.41 

97.80 

97.41 

97.74 

97.74 

97.69 

98.12 

98.63 

87.99 

Trees 

2405 

83.75 

71.03 

88.63 

78.44 

94.40 

90.32 

90.36 

89.77 

90.48 

92.51 

89.14 

Meadows 

1788 

93.34 

91.70 

94.08 

89.31 

94.20 

95.45 

95.62 

93.69 

94.08 

95.51 

93.52 

Bricks 

2140 

70.38 

70.95 

79.67 

75.36 

84.27 

85.97 

87.77 

86.02 

79.86 

85.59 

87.87 

Bare soil 

4677 

87.39 

92.47 

86.66 

92.45 

99.29 

99.31 

99.33 

99.01 

97.12 

99.29 

98.00 

Asphalt 

4844 

88.55 

89.26 

92.33 

92.85 

93.77 

94.91 

95.01 

92.48 

93.52 

95.10 

93.04 

Bitumen 

972 

78.98 

86.62 

84.82 

88.96 

93.63 

94.80 

94.80 

92.46 

82.48 

94.37 

92.57 

Tile 

1112 

71.44 

93.81 

82.72 

98.89 

98.06 

98.80 

98.80 

96.49 

97.41 

99.91 

100 

Shadows 

2020 

100 

98.64 

100 

99.80 

100 

100 

100 

100 

99.95 

100 

91.61 


TABLE II 

Global and Class-Specific Accuracies in Percentage for the Center of Pavia Image, Using the Morpho-MS Marker Selection 
Approach and Using the Previously Proposed Methods. The highest accuracies are bolded in each category. 
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- 

MV 

MV 
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97.14 
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AA 
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97.75 
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97.37 

98.13 

96.41 

96.41 

K 

97.39 

97.46 

97.32 

97.08 

97.62 

96.38 

96.38 

Water 

98.17 

97.93 

97.93 

97.98 

97.87 

98.27 

98.27 

Trees 

95.41 

94.27 

94.27 

94.19 

94.32 

91.62 

91.66 

Meadows 

95.79 

96.30 

96.30 

96.19 

97.33 

93.97 

93.97 

Bricks 

93.03 

94.88 

94.74 

89.86 

96.02 

96.82 

96.82 

Bare soil 

99.98 

99.94 

99.31 

99.63 

99.96 

98.92 

98.92 

Asphalt 

98.21 

98.96 

98.88 

99.02 

98.96 

95.64 

95.64 

Bitumen 

99.15 

97.98 

97.98 

99.47 

99.36 

93.74 

93.74 

Tile 

99.17 

99.45 

99.54 

100 

100 

98.80 

98.80 

Shadows 

100 

100 

100 

100 

99.40 

99.90 

99.90 



H Water 

■ Trees 

| | Meadows 

H Bricks 

| Bare soil 
I | Asphalt 
[ | Bitumen 

■ Tile 

I | Shadows 





Fig. 5. Center of Pavia image, (a) Three-band color composite, (b) Reference data, (c) M-HSEG op classification map (Proba-MS marker selection, SAM 
DC, S wg ht = 0.2). (d) MSF classification map (Proba-MS marker selection, SAM DC), (e) M-HSEG op classification map (Morpho-MS marker selection, 
SAM DC, S wght = 0.0). 
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A. Classification of Center of Pavia image 

The Center of Pavia image was acquired by the ROSIS 
optical sensor over the urban area of Pavia, Italy. The image 
has spatial dimensions of 785 by 300 pixels, with a spatial 
resolution of 1.3 m/pixel and 102 spectral channels. Nine 
classes of interest are considered, which are detailed in Table I, 
with the number of labeled samples for each class. Fig. 5 
shows a three-band false color image and the reference data. 
Thirty samples for each class were randomly chosen from 
the reference data as training samples. The remaining samples 
composed the test set. The training set was used for training an 
SVM classifier, while the test set was employed for estimating 
classification accuracies of the considered algorithms. 

The multiclass one-versus-one SVM classification, with the 
Gaussian radial basis function (RBF) kernel, of the hyperspec- 
tral image was performed. The optimal parameters C (penalty 
during the SVM optimization) and 7 (spread of the RBF 
kernel) were chosen by fivefold cross validation: C = 128, 
7 = 2 -5 . 

Then, marker selection was performed, using Proba-MS and 
Morpho-MS approaches. The Proba-MS method was applied 
with parameters M = 20, P = 40%. The threshold S was 
chosen to be equal to the lowest probability within the highest 
2% of the probability estimates for the whole image. These 
parameters were chosen (and applied for all three data sets) 
following the recommendations in [ 12 ], where it was shown 
that the method was robust to the selection of parameters. 

Finally, the M-HSEG segmentation of the images was 
performed, using the three proposed implementations. The LI 
norm and the SAM between the region mean vectors were 
applied as DCs 2 . In all experiments, the M-HSEG algorithm 
has been run until no more merging was possible. By assigning 
the class of each marker to the region containing this marker, 
the spectral- spatial classification maps were obtained. 

Tables I and II summarize the global and class-specific ac- 
curacies of the pixelwise SVM classification and the proposed 
method, using Proba-MS and Morpho-MS marker selection 
approaches, respectively, and S wg ht = [ 0 . 0 , 0 . 2 , 0.5]. The fol- 
lowing measures of accuracy were applied: Overall Accuracy 
(OA is the percentage of correctly classified pixels), Average 
Accuracy (AA is the percentage of correctly classified pixels 
for each class) and kappa coefficient k, [35]. In order to com- 
pare the results of the proposed method with other advanced 
classification techniques, we have included results obtained us- 
ing the construction of an MFS from the same sets of markers 
(see Section IV). Fig. 5 shows some of the corresponding clas- 
sification maps. We have also included in Table II accuracies 
of classification by majority voting within the neighborhoods 
defined by both watershed ( WH+MV ) and HSEG segmentation 
(. HSEG+MV , with the parameter S wg ht = 0 . 0 , which is 
equivalent to HSWO, and the SAM DC) [16]. The optimal 
hierarchical level for HSEG+MV was chosen interactively and 
validated by quantitatively evaluating classification accuracies 
of the considered method at several hierarchical levels versus 
the test data. In our previous works, we have shown that the 

2 The oo-norm was also applied as a DC; the corresponding results were 
not an improvement compared to the ones reported in this paper. 


MSF-based and the majority voting-based methods used here 
for comparison yield better accuracies when compared to the 
well known ECHO technique [9], [16] and the approach based 
on SVM and Extended Morphological Profiles (EMP) [11], 

[15]. 

The following conclusions can be drawn: 

• The proposed marker-based M-HSEG method yields ac- 
curate segmentation and classification results. The av- 
erage accuracy is improved by 2.9 and 5.2 percentage 
points when compared to the SVM classification, when 
applying the Proba-MS and Morpho-MS marker selection 
approaches, respectively. Therefore, it is useful to include 
markers in the HSEG algorithm, in order to automatically 
select the relevant segmentation level. 

• The M-HSEG op implementation significantly outper- 
forms M-HSEG r and M-HSEG P implementations in 
terms of accuracies. Thus, a region mean vector seems 
to be a “poor” representative feature of image regions. 
It may be advantageous to apply more complex models 
for describing each region and dissimilarities between 
regions, for instance, statistical region model proposed 
in [36]. 

• The M-HSEG op method performs similar with S wg ht = 
0.0 and S wg ht = 0.2 in terms of classification accuracies. 
However, classifications accuracies decrease with further 
increase of the S wg ht value. 

• The Morpho-MS marker selection approach yields the 
highest classification accuracies for this data set. 

• The proposed method using the Morpho-MS marker se- 
lection outperforms the WH+MV and HSEG+MV algo- 
rithms. 

• The M-HSEG op classification results are similar to the 
MSF-based results. However, all the DCs give compara- 
ble accuracies when applying the M-HSEG op approach. 
The MSF classification method using Proba-MS markers 
yields significantly lower accuracies when the SAM DC 
is applied. It assigns large portions of the water to 
the spatially adjacent asphalt regions, and assimilates 
shadows with neighboring regions (see Fig. 5(d)). Thus, 
the M-HSEG technique appears to be more robust when 
using different DCs. 

B. Classification of Indian Pines image 

The Indian Pines image is of vegetation area that was 
recorded by the AVIRIS sensor in Northwestern Indiana. It is 
of 145 by 145 pixels, with a spatial resolution of 20 m/pixel. 
Twenty water absorption bands have been removed [37], and 
a 200-band image was used for our experiments. Sixteen 
information classes are considered, which are detailed in 
Fig. 6 and in Table III, with the number of samples for 
each class in the reference data. Fifty samples for each class 
were randomly selected from the reference data to be used as 
training samples, except for classes “ alfalfa ”, “grass/pasture- 
mowed” and “oats”. These classes contain a small number 
of samples in the reference data. Therefore, only 15 samples 
for each of these classes were randomly chosen as training 
samples. The remaining samples composed the test set. 
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B Corn-no till 

|B Grass/trees 

B Corn-min till 

I | Grass/pasture-mowed 

| | Corn 

| Hay-windrowed 

B Soybeans-no till 

I | Oats 

| Soybeans-min till 

I | Wheat 

B Soybeans-clean till 

HI Woods 

| | Alfalfa 

| | Bldg-Grass-Tree-Drives 

B Grass/pasture 

[ | Stone-steel towers 




Fig. 6. Indian Pines image, (a) Reference data, (b) SVM classification map. (c) Proba-MS marker map. (d) M-HSEG op classification map (Proba-MS 
marker selection, SAM DC, S wg ht = 0-0)- (e) MSF classification map (Proba-MS marker selection, SAM DC), (f) Morpho-MS marker map. (g) M-HSEG op 
classification map (Morpho-MS marker selection, Inf DC, S wg nt — 0.0). 


TABLE III 

Results for the Indian Pines data set. Information Classes, Number of Labeled Samples, Global and Class-Specific Accuracies in 
Percentage, Using the Proba-MS Marker Selection Approach. The highest accuracies are bolded in each category. 



Number 
of label, 
samples 

M-HSEG r 

M-HSEG P 

M-HSEG op 


MSF 

DC 

SAM | 

| Inf 

SAM 

Inf 

SAM 

Inf 

SAM 

Inf 

SVM 

SAM 

Swght 

0.0 

0.2 

Overall Accuracy 


77.53 

76.63 

81.59 

81.16 

89.23 

89.00 

88.72 

89.01 

78.17 

89.65 

Average Accuracy 


84.54 

83.40 

87.09 

87.54 

93.44 

93.20 

93.40 

93.23 

85.97 

93.48 

Kappa Coefficient n 


74.48 

73.40 

79.10 

78.62 

87.72 

87.47 

87.15 

87.49 

75.33 

88.19 

Com-no till 

1434 

67.20 

66.84 

85.26 

84.47 

93.71 

92.05 

93.71 

92.05 

78.18 

96.10 

Corn-min till 

834 

74.62 

68.24 

64.41 

74.36 

73.98 

77.04 

73.85 

77.04 

69.64 

73.72 

Com 

234 

64.67 

59.78 

64.67 

67.39 

96.74 

95.11 

96.74 

95.65 

91.85 

95.65 

Soybeans-no till 

968 

71.57 

71.79 

83.66 

75.49 

99.02 

99.02 

99.02 

99.02 

82.03 

98.04 

Soybeans-min till 

2468 

68.16 

71.59 

68.90 

67.12 

75.68 

75.02 

74.52 

75.02 

58.95 

77.01 

Soybeans-clean till 

614 

85.82 

85.82 

87.77 

89.54 

96.10 

96.81 

96.10 

96.81 

87.94 

95.74 

Alfalfa 

54 

92.31 

92.31 

92.31 

92.31 

92.31 

92.31 

94.87 

92.31 

74.36 

94.87 

Grass/pasture 

497 

80.09 

82.55 

89.71 

92.17 

93.96 

93.74 

93.51 

93.74 

92.17 

94.18 

Grass/trees 

747 

77.19 

72.02 

83.07 

86.23 

97.85 

97.42 

97.85 

97.42 

91.68 

97.85 

Grass/pasture-mowed 

26 

100 

100 

100 

100 

100 

100 

100 

100 

100 

100 

Hay-windrowed 

489 

99.54 

99.54 

98.41 

99.54 

99.54 

99.32 

99.54 

99.32 

97.72 

99.54 

Oats 

20 

100 

100 

100 

100 

100 

100 

100 

100 

100 

100 

Wheat 

212 

97.53 

98.15 

98.15 

98.15 

99.38 

99.38 

99.38 

99.38 

98.77 

99.38 

Woods 

1294 

99.20 

92.28 

97.67 

94.05 

98.47 

98.47 

96.95 

98.47 

93.01 

98.39 

Bldg-Grass-Tree-Drives 

380 

76.97 

77.88 

83.94 

84.24 

82.73 

82.12 

82.73 

82.12 

61.52 

79.70 

Stone-steel towers 

95 

97.78 

95.56 

95.56 

95.56 

95.56 

93.33 

95.56 

93.33 

97.78 

95.56 


A pixelwise classification on the 200-band image was 
performed using the multiclass one-versus-one SVM classifier 
with the Gaussian RBF kernel and parameters chosen by 
fivefold cross validation: C = 128 and 7 = 2 -6 . Then, marker 
selection using the two proposed approaches was performed, 
with the same parameters as for the Center of Pavia image 
(see Fig. 6 ). Finally, M-HSEG classification was applied, with 
the SAM and Infinity norm DCs, and executed until no more 
merging was possible. 

Tables III and IV give the global and class-specific ac- 


curacies of the SVM classification and the proposed M- 
HSEG technique, using Proba-MS and Morpho-MS marker 
selection algorithms, respectively, and S wg ht = [ 0 . 0 , 0 . 2 ]. The 
performances of the proposed approach are compared with 
those obtained by constructing an MSF from the same sets of 
markers, using the SAM DC. Table IV also contains results 
of the WH+MV and HSEG+MV techniques, applied in the 
same way as for the Center of Pavia image. Some of the 
corresponding classification maps are depicted in Fig. 6 . 

From the tables, similar conclusions as for the previous 
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TABLE IV 

Global and Class-Specific Accuracies in Percentage for the 
Indian Pines IMAGE, USING THE MORPHO-MS MARKER SELECTION 
Approach and Using the Previously Proposed Methods. The 

HIGHEST ACCURACIES ARE BOLDED IN EACH CATEGORY. 



M-HSEG op 

MSF 

WH 

HSEG 

DC 

SAM | 

Inf 

SAM 

+ 

+ 

Swght 

0 . 

,0 


MV 

MV 

Overall Accuracy 

90.91 

91.05 

91.69 

86.63 

90.86 

Average Accuracy 

87.10 

87.36 

87.99 

91.61 

93.96 

Kappa coeff. n 

89.63 

89.79 

90.52 

84.83 

89.56 

Com-no till 

90.97 

91.55 

91.84 

94.22 

90.46 

Com-min till 

91.58 

91.33 

92.60 

78.06 

83.04 

Com 

98.91 

95.65 

95.65 

88.59 

95.65 

Soybeans-no till 

97.93 

96.51 

97.06 

96.30 

92.06 

Soybeans-min till 

82.13 

81.39 

82.34 

68.82 

84.04 

Soybeans-clean till 

96.63 

96.28 

96.28 

90.78 

95.39 

Alfalfa 

92.31 

92.31 

94.87 

94.87 

92.31 

Grass/pasture 

92.17 

92.17 

94.63 

95.08 

94.41 

Grass/trees 

96.84 

97.56 

97.85 

97.99 

97.56 

Grass/past.-mowed 

100 

100 

100 

100 

100 

Hay-windrowed 

99.77 

99.77 

99.77 

99.54 

99.54 

Oats 

0.00 

0.00 

0.00 

100 

100 

Wheat 

99.38 

98.15 

100 

99.38 

98.15 

Woods 

99.28 

99.60 

100 

97.11 

98.63 

Bldg-Grass-Tr.-Dr. 

57.88 

70.00 

69.39 

69.39 

82.12 

Stone-steel towers 

97.78 

95.56 

95.56 

95.56 

100 


data set can be derived. The global and most of the class- 
specific accuracies are significantly improved when compared 
to the SVM classification. The M-HSEG op performs the best 
among the three proposed implementations. It gives similar 
results with S wg ht = 0.0 and S wg ht = 0 . 2 . The average 
accuracy of the M-HSEG op classification is improved by 7.5 
and 1.4 percentage points when compared to the SVM results, 
when applying Proba-MS and Morpho-MS marker selection 
algorithms, respectively. 

The Morpho-MS marker selection approach yields higher 
overall, but lower average classification accuracies, when 
compared to the Proba-MS method. This is due to the fact 
that when applying the Morpho-MS technique, regions smaller 
than the size of the SE are not captured in a marker map, and 
thus they disappear in the final segmentation and classification 
maps. The corresponding classes to which these region belong 
(“oats” class) are not accurately classified. The Proba-MS 
method proves to be more robust when selecting markers for 
regions of different sizes. 

The MSF-based technique yields higher global accuracies 
when compared to the M-HSEG results for this data set, 
but the difference between average accuracies of both ap- 
proaches is less than 1%. The proposed M-HSEG op method 
yields significantly higher accuracies when compared to the 
WH+MV algorithm. Its average accuracy is only 0.5% lower 
when compared to the HSEG+MV technique, which gives the 
highest average accuracy, but where an appropriate level of 
segmentation detail was chosen interactively. As a conclusion, 
classification results obtained with the proposed marker-based 
HSEG are comparable to the ones obtained by using either an 
interactive approach or the test data for selecting an optimal 
level of segmentation detail from the HSEG results. 



Q Arable land M Green fallowland | Grassland | | Forest 

| | Row crops O Leguninosea | Reed 



(b) 


Fig. 7. Classification maps for the Hungarian image: (a) SVM map. (b) M- 
HSEG op map, using Morpho-MS marker selection, SAM DC, S wg ht = 0.0. 


C. Classification of Hungarian image 

The Hungarian hyperspectral image was recorded by the 
AISA Eagle sensor over the agricultural area of Heves, Hun- 
gary. The image is of 500 by 800 pixels, with a spatial 
resolution of 6 m/pixel. The number of spectral channels of the 
acquired image is 252 (with a spectral range from 395 to 975 
nm). The two most noisy channels were removed, and a 50- 
band image was used for experiments, obtained by averaging 
over every five adjacent bands [38]. The reference data contain 
seven classes of interest, detailed in Fig. 7 and in Table V, with 
the number of labeled pixels for each class. A hundred samples 
for each class were randomly chosen from the reference data 
as training samples, the remaining samples being used as the 
test set. 

The one-versus-one SVM classification with the Gaussian 
RBF kernel was performed, with the parameters chosen by 
fivefold cross validation: C = 128 and 7 = 2 -4 (see Fig. 7). 
Then, marker selection was applied in the same way as for the 
two previous data sets. Finally, both the M-HSEG and MSF 
classification were performed, using the SAM DC. Global and 
class-specific accuracies for the applied approaches, together 
with the WH+MV and HSEG+MV results, are given in 
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TABLE V 

Results for the Hungarian data set. Information Classes, Number of Labeled Samples and Classification Accuracies in 
Percentage, Using the SAM dissimilarity criterion: Overall Accuracy (OA), Average Accuracy (AA), Kappa Coefficient O) and 
Class-Specific Accuracies. The highest accuracies are bolded in each category. 


Marker selection 

Number 

Proba-MS 

Morpho-MS 


WH 

HSEG 

Classification 

of labeled 

| M-HSEG^ | 

MSF 

| M-HSEG up 

MSF 

SVM 

+ 

+ 

Swght 

samples 

0.0 

0.2 

- 

0.0 

0.2 

- 


MV 

MV 

OA 


79.86 

79.94 

80.40 

85.09 

84.72 

84.90 

77.36 

81.32 

85.78 

AA 


84.20 

84.25 

84.81 

89.36 

88.57 

88.60 

80.10 

84.53 

88.23 

n 


74.81 

74.91 

75.47 

81.24 

80.79 

80.98 

71.71 

76.52 

82.16 

Arable land 

10141 

67.77 

67.87 

68.43 

73.51 

72.99 

73.35 

66.60 

68.84 

74.92 

Row crops 

5602 

92.42 

92.48 

92.73 

96.18 

95.80 

95.93 

94.20 

96.08 

97.38 

Green fallowland 

4085 

87.25 

87.25 

87.93 

90.64 

91.17 

91.17 

83.44 

86.08 

90.24 

Leguninosea 

1031 

87.43 

87.43 

87.33 

91.41 

88.08 

86.79 

81.31 

88.35 

86.37 

Grassland 

2183 

83.29 

83.29 

83.58 

89.53 

89.53 

88.67 

78.40 

84.67 

87.41 

Reed 

5278 

80.05 

80.24 

80.55 

86.96 

86.75 

87.31 

73.77 

82.97 

89.44 

Forest 

771 

91.21 

91.21 

93.14 

97.32 

95.68 

97.02 

83.01 

84.74 

91.87 


Table V 3 . 

The obtained results are consistent with those obtained for 
the two previous images. All spectral- spatial classification 
results are more accurate when compared to the pixelwise 
SVM result. The best average accuracy is achieved using 
the Morpho-MS marker selection followed by the M-HSEG op 
classification, with S wg ht = 0.0. It is improved by 9.3 
percentage points when compared to the SVM classification. 
Fig. 7 shows the corresponding classification map, which 
contains more homogeneous regions when compared to the 
SVM result. The Hungarian image is composed of large 
regions mainly representing agricultural fields. Therefore, the 
Morpho-MS marker selection approach works very well for 
this data set, outperforming the Proha-MS technique in terms 
of accuracies. The M-HSEG op method with S wg ht = 0.2 
performs better than with S wg ht = 0.0 for the Proba-MS set of 
markers in terms of accuracies, but worse for the Morpho-MS 
marker map. 

As a conclusion, the experimental results on three data 
sets did show that the M-HSEG op with S wg ht = 0.0, i.e., 
best merge region growing controlled by classification-derived 
markers yields either the best or close to the best classifica- 
tion accuracies. Thus, it is recommended for spectral- spatial 
classification of hyperspectral images. 

VI. Conclusions 

The HSEG segmentation approach is one of the few state- 
of-art segmentation algorithms that both naturally exploits 
spectral and spatial information and produces a hierarchical 
set of image segmentations. Many application areas can greatly 
benefit from methods able to automatically analyze segmen- 
tation hierarchies and select a single optimum segmentation 
level. One of such application areas is remotely sensed hyper- 
spectral image classification, where segmentation can be used 
as a powerful tool for automatically defining both spatial and 
spectral dependencies in an image. 

In this paper, a new marker-based Hierarchical SEGmen- 
tation (M-HSEG) method for the automated selection of 

3 We have included in Table V only accuracies of the M-HSEG op imple- 
mentation of the proposed approach, which are significantly higher compared 
to those obtained using M-HSEG r and M-HSEG P implementations. 


a segmentation level from the segmentation hierarchy and 
hyperspectral image classification has been proposed. In this 
method, the automated choice of a single segmentation is 
achieved by using automatically selected markers. A marker 
map is first constructed using classification results. Two dif- 
ferent classification approaches for automatic marker selection 
for M-HSEG are adapted and compared. Then, the novel 
M-HSEG algorithm is applied, resulting in a spectral- spatial 
classification map. Three different ways of integrating markers 
into the HSEG technique are proposed and investigated. 

Experimental results, demonstrated on the three hyperspec- 
tral data sets, have shown that the new M-HSEG method 
yields accurate segmentation and classification maps and is 
sufficiently robust for classifying different kinds of images. Its 
performances are similar to the recently proposed MSF-based 
approach, which has been proven to outperform such standard 
spectral- spatial classification approaches as the ECHO and the 
SVM and EMP-based techniques. However, the proposed M- 
HSEG method has the following advantages when compared 
to the MSF-based technique: 

• At each iteration, the dissimilarity criterion is computed 
between every two regions, while in the case of the 
MSF construction the dissimilarity criterion is computed 
between two pixels. Thus, if a relevant feature vector for 
every region is chosen, with the increase of the number of 
pixels in the region its feature vector can better represent 
an information class. This advantage is important for 
classifying images containing classes and spatial regions 
with high internal variance. 

• M-HSEG provides a possibility of merging non-adjacent 
regions by clustering based on spectral information only. 
This may also lead to a more accurate and robust mod- 
eling and representation of classes of interest. 

• M-HSEG produces a segmentation hierarchy. Thus, it 
remains possible to further analyze segmentation maps 
at multiple levels of detail, in order to select the most 
relevant segmentation map for a specific application. 

It was concluded that a region mean vector feature may 
be not an accurate/sufficient representative of image regions. 
Therefore, in the future we plan to explore the choice of 
optimal representative features for segmentation regions, in 
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particular texture and shape features, for further improving 
segmentation and classification results. 
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